Bicriteria data compression

The advent of massive datasets and the consequent design of high-performing distributed storage systems—such as BigTable by Google [7], Cassandra by Facebook [5], Hadoop by Apache—have reignited the interest of the scientific and engineering community towards the design of lossless data compressors which achieve effective compression ratio and very efficient decompression speed. Lempel-Ziv’s LZ77 algorithm is the de facto choice in this scenario because its decompression is significantly faster than other approaches, and its algorithmic structure is flexible enough to trade decompression speed versus compressedspace efficiency. This algorithm has been declined in many ways, the most famous ones are: the classic gzip, LZ4 and Google’s Snappy. Each of these implementations offers a trade-off between space occupancy and decompression speed, so software engineers have to content themselves by picking the one which comes closer to the requirements of the application in their hands. Starting from these premises, and for the first time in the literature, we address in this paper the problem of trading optimally, and in a principled way, the consumption of these two resources by introducing and solving what we call the Bicriteria LZ77-Parsing problem. The goal is to determine an LZ77 parsing which minimizes the space occupancy in bits of the compressed file, provided that the decompression time is bounded by a fixed amount. Symmetrically, we can exchange the role of the two resources and thus ask for minimizing the decompression time provided that the compressed space is bounded by a fixed amount. This way, the software engineer can set its space (or time) requirements and then derive the LZ77 parsing which optimizes the decompression speed (or the space occupancy, respectively), thus resulting the best possible LZ77 compression under those constraints. We solve this problem in four stages: we turn it into a sort of weight-constrained shortest path problem (WCSPP) over a weighted graph derived from the LZ77-parsing of the input file; we argue that known solutions for WSCPP are inefficient and thus unusable in practice; we prove some interesting structural properties about that graph, and then design an O(n log n)-time algorithm which computes a small additive approximation of the optimal LZ77 parsing. This additive approximation is logarithmic in the input size and thus totally negligible in practice. Finally, we sustain these arguments by performing some experiments which show that our algorithm combines the best properties of known compressors: its decompression time is close to the fastest Snappy’s and LZ4’s, and its compression ratio is close to the more succinct bzip2’s and LZMA’s. Actually, in many cases our compressor improves the best known engineered solutions mentioned above, so we can safely state that with our result software engineers have an algorithmic-knob to automatically trade in a principled way the time/space requirements of their applications. Summarizing, the three main contributions of the paper are: (i) we introduce the novel Bicriteria LZ77-Parsing problem which formalizes in a principled way what data-compressors have traditionally approached by means of heuristics; (ii) we solve this problem efficiently in O(n log n) time and optimal linear space, by proving and deploying some specific structural properties of the weighted graph derived from the possible LZ77-parsings of the input file; (iii) we execute a preliminary set of experiments which show that our novel proposal dominates all the highly engineered competitors, hence offering a win-win situation in theory&practice. ar X iv :1 30 7. 38 72 v1 [ cs .I T ] 1 5 Ju l 2 01 3